On the non-stationarity of financial time series: impact on optimal portfolio selection 
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We investigate the possible drawbacks of employing the standard Pearson estimator to measure 
correlation coefficients between financial stocks in the presence of non-stationary behavior, and we 
provide empirical evidence against the well-established common knowledge that using longer price 
time series provides better, more accurate, correlation estimates. Then, we investigate the possible 
consequences of instabilities in empirical correlation coefficient measurements on optimal portfolio 
selection. We rely on previously published works which provide a framework allowing to take into 
account possible risk underestimations due to the non-optimality of the portfolio weights being 
used in order to distinguish such non-optimality effects from risk underestimations genuinely due to 
non-stationarities. We interpret such results in terms of instabilities in some spectral properties of 
portfolio correlation matrices. 
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I. INTRODUCTION 



Ever since the fundamental work by Markowitz 1], the study and empirical analysis of correlations between stocks 
traded in a financial market have represented topics of paramount importance in financial analysis. In a nutshell, this is 
because standard optimal portfolio selection theory (OPST) heavily relies on the knowledge of the portfolio correlation 
matrix. From the physicist's viewpoint, this poses a measurement problem. Namely, OPST works perfectly whenever 
the true correlations between the stocks forming a portfolio are known. However, such quantities are unobservable by 
definition, and what one usually does is to measure the correlation coefficient between two stocks i and j according 
to the well-known Pearson estimator: 

o 



Pij = j;^2 r it r 3t, (1) 

t=l 



where ra (here assumed to be standardized, i.e. zero mean and unit standard deviation) denotes the price change 
of stock i over the time step from t — 1 to t, for t = 1, . . . , T. It is intuitively clear, and can be rigorously shown, 
that using longer time series, i.e. exploiting more information, will lead to better correlation estimates, so that in the 
limit of infinitely long time series the Pearson estimator will converge to the true correlation coefficient pij between 
stocks i and j: pij — > pij for T — > oo. Of course, in real financial applications one always needs to cope with time 
series of finite length. So, when dealing with portfolio selection, being able to make quantitative statements on the 
reliability of the empirically measured correlation coefficients is highly desirable. In particular, it is relevant to assess 
the level of measurement noise which might affect correlation estimates and consequently to devise possible filtering 
techniques to amend empirically observed correlation matrices from noise. Ever since the pioneering works [2|, |3[, the 
Econophyiscs community identified random matrix theory (RMT) as a valuable tool to understand the correlation 
structure of financial markets (or portfolios) in terms of spectral properties of correlation matrices, shedding light 
on several interesting stylized facts (see for example [4( for a review) and providing rigorous mathematical relations 
between a true correlation matrix and its Pearson estimators Q . 
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The vast majority of the aforementioned results were derived under the assumption that the group of stocks whose 
correlations are to be measured is described by a joint stationary probability distribution. Loosely speaking, this 
amounts to assuming that a true, constant, correlation coefficient pij between each pair of stocks i and j actually 
exists and can be measured, therefore motivating the previously mentioned intuitive notion that employing longer 
time series provides better correlation estimates. The goal of this paper is to challenge this common knowledge, 
providing some empirical evidence against it, and to eventually provide a quantitative analysis of the impact that 
non-stationarities might have on portfolio risk. In recent years, other authors tackled similar problems from different 
viewpoints. For example, in 0, HJ a phenomenological model was proposed in order to study the feedback mechanism 
according to which correlations determine optimal portfolios but are simultaneously affected by investments based on 
them. Even more recently, different correlation structures induced by non-stationarities in financial time series were 
exploited to identify and categorize the possible states of a financial market [9(. On a more practical level, it is worth 
mentioning that the RiskMetrics risk management tool [Io| . freely available since 1992, introduced a first systematic 
way to take possible non-stationarities into account by exponentially dampening the contribution of older prices to 
correlation estimates. 

Our approach in this paper will be rather evidence-based, and we shall not try to investigate the possible causes 
for the emergence of non-stationarities in financial dynamics. Our first goal will be to explore the consequences of 
possible non-stationarities on empirically measured correlation coefficients between pairs of stocks. More specifically, 
it is reasonable to expect the distributional properties of a given stock to change over time, especially when considering 
large time horizons. It is also reasonable to expect different stocks to change at different paces. All in all, the combined 
effect of rather diverse distributional changes happening at different "velocities" will impact the correlations between 
stocks. In the following we shall look for empirical evidence in this direction, both in a global and local sense. First, 
we shall consider correlation coefficient measurements (between pairs of stocks) performed over non-overlapping time 
windows, and we shall perform a statistical test to check their mutual, global, compatibility. Then, we shall test 
the local compatibility among consecutive correlation estimates performed over increasingly large time windows. We 
anticipate here that both tests will reveal substantial violations of the null stationarity hypothesis, and this will 
motivate our interest in studying the way that such effects might impact the correlation structure of a portfolio and 
its overall risk. 

The paper is organized as follows. The results of the aforementioned global and local stationarity tests, performed 
on two financial datasets, will be detailed and discussed in Sections HT1 and Hill respectively. In Section HV1 the effects 
of inconsistencies in correlation coefficient measurement on portfolio risk assessment will be presented, and a possible 
interpretation of the results will be given in terms of instabilities in the correlation matrix eigenvalue spectrum. The 
conclusions and outlook of this work will then be discussed in Section [V] 



II. GLOBAL STATIONARITY 



The probability distribution of the measured correlation coefficient p between a pair of random variables X and Y 
described by a bivariate Gaussian probability density is given by [111 ]: 

F(p;p,T) = i(T-2)(f-p 2 r- 4 )/ 2 (f-p 2 r- 1 )/ 2 / + °° dr 1 , P 6[-l,l]. (2) 

7T J (cosh r- ppY 1 

In this expression p G [— 1, 1] is the true correlation coefficient between X and Y, i.e. (we denote the expectation 
with respect to the Gaussian joint probability density describing X and Y as Mq[. . .]) 



E G [(X-E G [X])(Y-E G [Y})} 



^E G [(X - E G [X])*]Ea[(Y - E G [Y]) 2 ] 
whereas p is the corresponding Pearson estimator over a sampling time window T (see also equation (JTJ), i.e. 



t=l 

where x t and y t (t = 1, . . . , T) are the time-i standardized observations of X and Y. In Figure Q] some examples of the 
probability density in equation @ are plotted for different values of p and T. It can be shown that the probability 
density @ has mean and variance given by 
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FIG. 1: Left: A few examples of the probability density in equation J2| for T = 50 (dashed lines) and T = 150 (continuous 
lines). Both sets of curves refer to the values p = 0, ±0.3, ±0.6, ±0.9. As one would naturally expect, it can be seen that, for 
the same value of p, curves referring to a smaller sample size (i.e. a smaller T) are broader than their counterparts obtained 
with a larger T. Right: comparison of the probability density in equation ([2]) (with T — 50 and p = 0.2) and its Gaussian 
approximation (dashed line). 



E P lp}=P~^g^ (5) 

E P [(P-E P [ P ]) 2 ] = ^(l + f + ...), (6) 

and, already for reasonably large samples, one can realize that such quantities can be safely approximated as mp = p 
and ap = (1 — p 2 ) 2 /T, respectively. Moreover, it is easy to verify that, for sufficiently large sample dimensions T, the 
density ^ is fairly well approximated by a Gaussian distribution with mean mp and standard deviation ap (see for 
example the right plot in Figure [T]for a comparison between the two densities with T = 50 and p = 0.2), the price to 
pay being an "unphysical" support covering the whole real axis. 

The probability density of measured correlation coefficients in equation (J2|) is derived under the assumption that 
the random variables X and Y are described by a stationary joint probability density. In other words, a consequence 
of this assumption (when verified) is that different Pearson estimates (see equation (JTJ) computed over independent, 
i.e. non-overlapping, time windows of the same length T will be distributed according to equation (J5|). For this very 
reason, any empirical violation of the probability distribution in equation @ hints at possible non-stationarities in 
the joint distribution of the random variables X and Y, which reflect into non-stationarities in their observed time 
series Xt and yt- Therefore, a global test of stationarity for financial time series can be performed by comparing 
the empirical distribution of measured correlation coefficients (over non-overlapping time windows) between a given 
pair of stocks and the probability density in equation (J2J), seen as a "null" stationarity hypothesis, where we assume 
pij to be equal to the correlation coefficient measured over the union of all the non-overlapping time windows. We 
performed such a test on two financial datasets made of N$ = 412 stocks belonging to the American S&P500 Index 
and Np = 137 stocks belonging to the British FTSE350 Index, respectively. Both datasets are made of 1758 daily 
price changes covering the years 2005-2011. For each pair of stocks in both datasets (Ns(Ns — l)/2 = 84666 pairs 
for the S&P dataset, Np(Np — l)/2 = 9316 pairs for the FTSE dataset) the empirical distribution of correlation 
coefficients estimated over time windows of different lengths was compared to the probability density in equation ([2]) 
by performing a Kolmogorov-Smirnov (KS) test [121 ] . The results are summarized in Table[I] as can be seen, especially 
in the S&P dataset, a relevant fraction of the available stock pairs is found to violate our global stationarity test. In 
particular, larger violations occur for smaller sample sizes (in our case T = 25, roughly corresponding to one trading 
month): for example, over such sample size we find a remarkably high fraction (8.49%) of stock pairs belonging to 
the S&P500 which do not fit the stationarity hypothesis even when considering the rather restrictive significance level 
a = 0.01. 

We are by all means aware that performing the tests we have detailed with the probability distribution in equation 
@ amounts to a certain level of approximation, at least for two reasons. First, equation is derived under the 
assumption of normally distributed data, whereas it is very well-known that daily financial data are usually much 
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TABLE I: Results of the KS test comparing the empirical distribution of correlation coefficients measured over non-overlapping 
time windows of length T to the theoretical one in equation We consider significance values a = 0.01,0.05,0.1 and time 
windows of length T = 25, 50, 100. For each value of T and for each significance level a the fraction of stock pairs violating 
the null stationarity hypothesis is reported, alues between parentheses refer to the fractions of stock pairs still violating the 
test after a random reshuffling is performed in each time series (see the main text for more details). Given that both our S&P 
and FTSE datasets are made of 1758 returns, when considering samples made of 25 daily returns we obtain 70 correlation 
coefficient estimates over non-overlapping time windows, when considering T — 50 we have 35 estimates and for T = 100 we 
have 17 estimates. Results of the same test (for a = 0.05) are also reported for Student-t distributed Monte Carlo data having 
the same correlation structure and dimensions of the two financial datasets we use (see the main text for more details). 



S&P 


T = 25 


T = 50 


T = 100 


a = 0.01 


8.49% (1.87%) 


4.43% (0.5%) 


3.39% (0.3%) 


a = 0.05 


27.34% (9.29%) 


21.04% (4.94%) 


18.99% (1.81%) 


a = 0.10 


42.80% (25.67%) 


37.16% (10.81%) 


34.45% (4.57%) 


S&P (Monte Carlo) 


T = 25 


T = 50 


T = 100 


a = 0.05 


0.9% 


1.1% 


0.8% 


FTSE 


T = 25 


T = 50 


T = 100 


a = 0.01 


2.04% (0.5%) 


1.28% (0.3%) 


0.6% (0.02%) 


a = 0.05 


10.40% (3.67%) 


9.35% (2.82%) 


7.01% (1.5%) 


a = 0.10 


20.03% (9.35%) 


19.73% (4.33%) 


16.88% (2.80%) 


FTSE (Monte Carlo) 


T = 25 


T = 50 


T = 100 


a = 0.05 


0.8% 


0.3% 


0.3% 



more heavy-tailed (typically displaying power law tails with exponent 3 / 3.5). Second, equation ([2]) requires the 
hypothesis that the random variables under study are described by a multivariate Gaussian distribution, so that a 
pair of variables has a bivariate Gaussian distribution. This is not exactly the case for financial price changes. Thus, 
in order to validate the statistical relevance of the results presented in Table Q] we generated two Monte Carlo datasets 
distributed according to a Student-t distribution having tail exponent equal to 3 and having the same dimensions 
(both in N and T) and correlation structure (over the whole sampling time) of our financial datasets. The results of 
our global stationarity test, when performed on such synthetic data, are also presented in Table U for the significance 
level a — 0.05. As the reader can immediately see, we find very few pairs failing the stationarity test: this evidence 
essentially rules out the heavy-tailed nature of the data and their distributional properties as possible causes for the 
very large fraction of stock pairs failing the test. It is then safe to state that non-stationarities are indeed the main 
failure cause in such a test. 

As a further check of the statistical significance of the results presented in Table HI we also performed the same 
global stationarity test on our two datasets after performing a synchronous random reshuffling. More precisely, in 
each of the two datasets we performed one same random reshuffling of the price changes in each time series. Such 
an operation clearly leaves the overall cross-correlation structure of the dataset intact, whereas it destroys (most of) 
the correlation dynamics in the data. In other words, this type of reshuffling is supposed to generate a stationary 
dataset. The results we obtained from our global stationarity test after performing the reshuffling are reported in 
Table U within parentheses. As one can see, for each combination of the sampling time T and the significance level a, 
the fraction of stock pairs failing the test is significantly reduced when the reshuffling is performed. 

In the following Section we shall try to motivate these findings by investigating more "local" properties of empirically 
measured correlation coefficients. 



III. LOCAL STATIONARITY 

In Figure [5] a possible qualitative explanation for the evidence presented in the previous Section (see Table Q] in 
particular) is pictured by plotting the "time evolution" of the correlation coefficient estimates between four stocks 
(six correlation coefficients) belonging to the S&P dataset. More specifically, starting from a first estimate computed 
over the first 200 days of the 1758 making the dataset, correlation coefficients among price changes are recomputed 
over increasingly large time windows by adding 10 days each time. As can be seen in the left plot of Figure [21 the 
correlation estimates display a rather wild fluctuating behavior, and, for most of the time, have actually very little to 
do with the estimates computed over the whole period of 1758 days (shown as straight lines in the plot). If a constant, 
or at least approximately constant, true value of the correlation coefficient between two stocks actually existed, then 
the estimation process would move on as in the right plot of Figure [3J which was produced with simulated data having 



5 




t (days) t (days) 



FIG. 2: Both plots show the "time evolution" of correlation estimates computed over increasingly large time windows. The left 
plot refers to the correlation coefficients computed for a group of four stocks belonging to our S&P dataset, while the right plot 
refers to Monte Carlo data generated with the same correlation structure of the stocks in the left plot. Correlation coefficient 
estimates computed over the whole available time window (1758 days) are shown as horizontal lines. 



the same correlation structure of the four stocks in the left plot. In this case, a true correlation coefficient does exist, 
and, as a matter of fact, estimates obtained from larger samples tend to lie closer to such a true value, progressively 
reducing the fluctuations around it (an infinitely long time series would eventually lead the correlation coefficient 
estimates to converge exactly to their corresponding true value). In this respect, the two plots in Figure [5] drastically 
defy the common knowledge that longer time series should eventually produce better correlation estimates. 

It is easy to verify that the statistical error on the correlation coefficient estimates introduced in equation ([1]) scales 
as T~ x / 2 . Thus, a local stationarity test can be performed as follows. Let us divide the whole sampling interval T 
into K equal parts of size r, i.e. T = Kt. So, given the price changes ru and r Jt of two stocks, let us estimate the 
correlation coefficient between them up to the K-th interval (k = 1, . . . , K — 1) by specializing equation (JTJ) as follows: 



t=i 

Then, one can compute the estimate p K +i over the next interval. Since the error on p K is of order a K = 1/^/kt, one 
can then assess the compatibility of consecutive correlation estimates between the stocks i and j by checking whether 
Pk+i € [pK ~ n &K, Pk + na K ] for some integer n or not. 

The aforementioned local stationarity test was performed on the previously introduced S&P and FTSE datasets. 
In particular, for each pair of stocks i and j we computed a first correlation estimate over T\ < T days, and then 
divided the remaining Ti = T — T\ days into K chunks of length r, as in the previous example. The following 
cases were considered: T\ = 200 and r = 50 (giving 32 correlation estimates for each pair of stocks), T\ = 200 and 
t = 100 (16 estimates), T\ = 250 and r = 250 (7 estimates). The results are summarized in Table HI1 the test was 
performed, for all of the aforementioned values of the lag r, by counting the fraction of all correlation estimates 
computed over the (k + l)-th lag falling outside the interval [p K — na K , p K + no~ K ], for n = 1, 3, 5. As one can see, we 
find relevant fractions of estimates violating our local stationarity test, especially when considering r = 250, roughly 
corresponding to one trading year. In this latter case we find a remarkable 1.20% of the S&P estimates and 1.04% 
of the FTSE estimates failing the test even when n = 5, meaning that year-to-year correlation estimates between 
some stock pairs can exhibit unexpectedly huge fluctuations. In analogy to what we did in the previous Section (see 
Table H|, we verified the statistical relevance of our findings by performing the local stationarity test on two sets 
of synthetic Student-t distributed data generated with the same correlation structure (over the whole time window 
T = T\ + T2) of our original S&P and FTSE datasets. As can be seen in Table ITT1 the fraction of correlation estimates 
failing the test is significantly reduced when considering Monte Carlo data. This result essentially rules out the data 
distributional properties, such as heavy tails, as possible causes for failing the local stationarity test, leaving us with 
non-stationarities as the main possible cause. 
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TABLE II: Results of the local stationarity test described in the main test. For each dataset (S&P and FTSE) we report 
the number of correlation coefficient estimates failing the test for different confidence interval amplitudes characterized by 
n = 1,3, 5. We also report the same results for two Monte Carlo datasets generated with the same correlation structure of the 
original financial datasets. 



S&P 


r = 50 


r = 100 


r = 250 


n — 1 


8.69% 


1 Q 32% 


40 39% 


n — 3 


1.00% 


2.75% 


8.85% 


n — 5 


0.1% 


0.3% 


1.20% 


S&P (Monte Carlo) 


r = 50 


T = 100 


r = 250 


n — 1 


1 1% 

J L /0 


9 74% 


7 4^% 

( .4:0 /0 


n — 3 


0% 


0.1% 


0.2% 


n = 5 


0% 


0% 


0% 


FTSE 


r = 50 


r = 100 


r = 250 


n = 1 


9.52% 


21.46% 


39.65% 


n — 3 


0.7% 


2.37% 


5.74% 


n — 5 


0.2% 


0.5% 


1.04% 


FTSE (Monte Carlo) 


r = 50 


T = 100 


r = 250 


n = 1 


1.06% 


8.96% 


17.03% 


n — 3 


0% 


1.02% 


0.7% 


n = 5 


0% 


0% 


0% 



IV. OPTIMAL PORTFOLIO SELECTION 



According to OPST, due to Markowitz [l|, the optimal weights for a portfolio built out of a given set of stocks can 
be determined explicitly in a number of different situations. Let us assume that our portfolio has to be put together 
as a combination of N stocks, and let us indicate, as already done throughout the paper, the time-t price changes of 
such stocks as r^. Thus, assuming we hold a quantity Wi of stock i over the time interval [t — 1, t], the portfolio value 
change over such interval is simply given by 



JV 

II t = ^ Wl r It . (8) 

i=i 

Usually, Markowitz 's OPST is formulated in terms of portfolio return maximization for a given fixed risk level, 
quantified by the portfolio variance, or risk minimization for a fixed expected return. The portfolio variance is given 
by 



N N 

on = Wiwfi l3 = ^ WiWjViUjPij, (9) 

where Cy = utajPij is the covariance matrix element between stocks i and j, whereas <Ji is the standard deviation 
of r, and pij is, as usual, the correlation coefficient between stocks i and j. Following [l3j], one can simply choose to 
minimize portfolio risk under a budget constraint: 



JV 



= i. 



(10) 



So, all in all, this amounts to solving the following optimization problem: 



d 



N 



3=1 



= 0, (11) 
where u>* denotes the optimal weight on stock i and £ is a Lagrange multiplier. Quite straightforwardly, one finds 
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(12) 



and this result shows that optimal portfolio weights heavily depend on correlations through the covariance matrix 
Cij — OiOjPij. Incidentally, let us remark here that the above construction actually implies T > N, where T is 
the number of recorded price changes for each stock. In a nutshell, this is because the eigenvalue spectrum of the 
covariance matrix C yields min{iV, T} non-zero eigenvalues. Now, since C is N x N, this means that if one had 
T < N, then N — T zero modes would appear in the spectrum, and, given the dependence of the w*s on the inverse 
of the covariance matrix, this would make the optimal portfolio problem ill-defined. 

As discussed in [l3T - ll5l |. different notions of risk can be introduced from the optimal weights in (|12p. depending on 
the covariance matrix being used. First of all, assuming correlations and volatilities to be constant over time, one can 
define the true optimal risk of the portfolio II: 



T\2 



On) 



{w 



(13) 



where C T (with entries Cy) is the true portfolio covariance matrix, whereas (wj)* denotes the corresponding optimal 
weights. Of course, the true risk of a portfolio cannot be known, and the only thing one can do is to compute the 
optimal weights starting from a covariance matrix estimate C E , computed over some interval T. So, the estimated 
risk from a given sample reads 



E^2 



(on) 



JV 

E 



ci (wfnwfy 



(14) 



This type of risk can also be meaningfully called in-sample risk, since it only provides risk estimates over past time 
intervals. What can actually be done in practice in order to infer future risk levels is to estimate the covariance 
matrix over some time interval T\, compute the corresponding optimal weights via equation (|12p . and then retain 
such weights over a consecutive time interval Ti- This is called realized risk, and it reads 



N 

On) 2 = E (15) 

where 10^ i is the optimal weight computed over time interval Tj., while C^ 2 (with entries C|3 2 y) is the covariance 
matrix over the following time interval Ti . So, all in all, the main idea is to use the in-sample risk computed over T\ 
as a proxy for the realized risk over T2- In references [1 31 ] the ratio 



q = og/og, (16) 

which gives information on possible underestimations of the realized risk, was extensively studied by means of Monte 
Carlo simulations (so that also the true portfolio risk (| 13[) was known and under control). A fundamental and intuitive 
result presented in [l3| is that the q ratio (fT()]) is always bigger than one. This is due to the fact that cr^j is a genuinely 
optimal risk, whereas is not, being computed with the weights i; which are not optimal with respect to the 
covariance matrix C^ 2 . Another, much less intuitive, result presented in [l3[ is that the q ratio is essentially model- 
independent. In other words, whenever the number of stocks N and the sampling intervals T\ and T% are kept fixed, 
the q ratio and its sample-to-sample fluctuations show no real dependence on the type of model used to define the 
true covariance matrix C T , which obviously determines the nature of its estimators Cf and Cf. See also [IM1 for 
similar discussions on related topics. 

The framework outlined above allows to quantify, via the q ratio in equation (|16[) . the risk underestimation being 
introduced whenever past optimal weights are retained over future time intervals. Moreover, as already pointed out, 
even the statistical error on q is model-independent up to very good approximation. So, whenever N, T\ and T2 
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FIG. 3: Comparison between the q ratios (equation l|16| l) obtained with real portfolio data (purple crosses) and the corresponding 
5-sigma confidence level intervals obtained under the hypothesis of stationary data (light blue for uncorrelated Gaussian data 
and dark blue for Gaussian data with the same correlation structure of the empirical data in use). 



have been fixed, one can obtain a very accurate confidence level interval for q by performing a portfolio Monte Carlo 
simulation with statistics large enough to reach the desired accuracy. We relied on these results in order to check 
whether secondary effects due to non-stationarities might lead to further risk underestimations, i.e. to unexpectedly 
large values of the q ratio in (|16l) . In Figure [3] we report examples of the results we obtained. We randomly selected 
N = 80 stocks both in the S&P and FTSE datasets, and used them to form portfolios (we checked that random stock 
selection does not impact the overall qualitative appearance of the results). Then, in the S&P case we divided the 
time series for the N selected stocks into chunks of length T\ = T% = 150 days, using the first T\ days for determining 
optimal portfolio weights, and then retaining them over the second period of T2 days. We did the same thing for the 
FTSE dataset using T\ = T2 = 100 days in that case. In order to work with as many samples as possible, we employed 
the T2 days used to compute the realized risk of the n-th portfolio as the T\ days used to compute the in-sample risk 
of the (n + l)-th portfolio. The dots and error bars in Figure [3] refer to Monte Carlo simulations of 100 synthetic 
portfolios generated with the same covariance structure (dark blue) of the data they are compared to and with no 
correlation at all (light blue). The dots represent estimates for the q ratio, whereas bars represent 5-standard deviation 
intervals (incidentally, one can sec there is almost no difference between the two intervals, due to the aforementioned 
model-independence of the q ratio and its error). On the other hand, purple crosses represent q ratios computed with 
real data. The plot on the left is obtained from S&P data, whereas the plot on the right refers to FTSE data. As can 
be seen, especially in this latter case, surprisingly large risk underestimations can happen, well outside the range one 
would have simply because of the non-optimality of the portfolio weights being held over From now on we shall 
refer to such region as to the non-optimality region. In the following we shall see how this evidence can be interpreted 
by investigating the spectral properties of portfolio correlation matrices. However, before we do so, let us clarify 
a subtle (yet rather important) point. Back in Section UH it was shown that correlation estimates computed over 
non-overlapping time windows are not always compatible with each other, and this result was essentially interpreted, 
in Section Hill in terms of time series non-stationarities and the consequently induced correlation dynamics (portrayed 
by the left plot in Figure [2]) . We commented these observations by remarking that such evidence goes against the 
intuitive notion that measuring correlations over very long time series produces better estimates. However, such a 
remark is not to be confused with another effect related to the use of time series of different length when assessing 
portfolio risk. Such an effect is portrayed in Figure [4) where the q ratio and its error are evaluated for the same 
S&P portfolio used to produce the left plot in Figure [3] with T 2 = 150 and I\ = 100, 125, 150 (red, blue and purple 
lines, respectively). As one can immediately recognize, in this case employing longer time series generally reduces 
the q ratio and its statistical uncertainty. This means that employing longer time series reduces the possible risk 
underestimations due to the non-optimality of the portfolio weights being used. However, it is important to stress 
that this does not necessarily protect against unexpectedly large fluctuations due to non-stationarities, as exemplified 
by sample 4 in Figure 0] which violates the non-optimality bounds for all three values of T\ . 
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FIG. 4: q ratio for the same portfolio used to produce the left plot in Figure [3] For all samples we have Ta = 150 days. The 
lengths of the sampling interval Ti are 100 (red lines), 150 (blue lines) and 200 (purple line) days. 

A. Spectral analysis 

Given a set of N stocks and two consecutive sampling periods T\ and T2, let us introduce the N x N correlation 
matrices p^ % and p® 2 with entries pj^ y and p^ 2 y representing the estimated Pearson correlation coefficient between 
the price changes of stocks i and j over the sampling periods T\ and T 2 , respectively. Correlation matrices are 
symmetric and positive definite by construction. Thus, we can introduce the eigenvalues < \r m ,i _• •Vr m ,2 < ■ ■ ■ < 
^T m .N (where to = 1,2 denotes the sampling period) of the two matrices pj, and p^ 2 introduced above, and the 

corresponding normalized eigenvectors vr m .„ = («^ „,-••, n ) (where to = 1, 2 and n = 1, . . . , N). Now, ever 
since the works 0, Q it has been very well known that the eigenvalue spectrum of a financial correlation matrix 
typically displays a main bulk of small eigenvalues plus a few, much larger, eigenvalues "leaking out" of such bulk. 
As a first approximation, the former can be identified as the noisy part of the spectrum, whereas the latter can be 
seen as the only eigenvalues carrying meaningful information on the correlation structure of the market or portfolio 
under analysis (see [19] , where a more refined analysis of the information amount actually carried by the bulk and 
the larger eigenvalues was carried out). Moreover, taking a closer look at the large eigenvalues (for example by 
means of principal component analysis (PCA)) unravels the following properties. The largest eigenvalue \T m ,N is 
usually of order AT, and it emerges as a consequence of global fluctuations involving all the stocks in the market or 
portfolio. Thus, it is generally labeled as the market eigenvalue. On the other hand, the next few largest eigenvalues 
At,„,tv-i, -^T m ,N—2 1 • • ■ ^T m ,N-s (usually for S ~ 3 — 5) typically arise from fluctuations involving only a certain fraction 
of the N stocks under consideration, and such stocks are found to belong to the same market sector in most cases. 
For this very reason, the eigenvalues \r m ,N-i, ^T m ,N—2, ■ ■ ■ , ^T m ,N-s are generally labeled as sector eigenvalues. 

Such evidence can be often unraveled by means of inverse participation ratios (IPRs). Given an N— dimensional 
normalized vector w = (w^, . . . , v)( N '), its IPR I w is defined as the sum of the fourth power of its components: 

/w = E( W «) 4 . (17) 

i=i 

So, IPRs can essentially be regarded as a tool to highlight the localization properties of a normalized vector. As a 
matter of fact, a fully delocalized vector with all components equal to 1/%/iV has IPR equal to 1/N, while a completely 
localized vector with one component equal to one and all others equal to zero has IPR equal to one. Moreover, PCA 
tells us that the generic price change rx m ,i(t) (for m = 1,2, i — 1, . . . , N and time t € (1, Ti] for to = 1 or t € [T+ 1,T] 
for to = 2) can be decomposed as 

N 

r Tm ,i(t) = V x T m ,e vf mA e Tm ,t(i), (18) 
i=\ 

where &T m ,t denotes the ^-th principal component. Equation (|18p shows that each eigenvector enters the dynamics of 
each stock. Therefore, a very delocalized eigenvector, as the one typically related to the market eigenvalue, will make 
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TABLE III: Spectral quantities of interest (market eigenvalues, sum of the first few sector eigenvalues and IPR of the market 
eigenvector) computed from the correlation matrix of a portfolio made of N = 80 stocks belonging to the S&P dataset. The 
sampling times 74 and T2 are both equal to 150 days. 



Sample 






A Tl 


At 2 


/ VTi ,„(xio- 2 ; 


) 7 VT . 2 , N (xl0- 2 ) 


1 


19.91 


17.20 


11.29 


11.90 


1.51 


1.55 


2 


17.20 


20.10 


11.90 


12.43 


1.55 


1.48 


3 


20.10 


19.47 


12.43 


10.42 


1.48 


1.47 


4* 


19.47 


30.15 


10.42 


9.46 


1.47 


1.44 


5 


30.15 


27.81 




12.88 


1.44 


1.55 


6* 


27.81 


43.87 


12.88 


9.40 


1.55 


1.32 


7 


43.87 


36.11 


9.40 


9.87 


1.32 


1.44 


8 


36.11 


31.91 


9.87 


7.86 


1.44 


1.42 


9 


31.91 


45.30 


7.86 


5.76 


1.42 


1.31 


10 


45.30 


28.44 


5.76 


9.61 


1.31 


1.42 


TABLE IV: Spectral quantities of interest (market 


eigenvalues, 


sum of the first few sector eigenvalues and IPR of the market 


eigenvector) computed from the correlation matrix of a portfolio made of N = 8 


stocks belonging 


to the FTSE dataset. The 


sampling 


times Ti and T2 


are both equal to 100 days. 








Sample 


AtlJV 


At 2 ,JV 


A Tl 


At 2 


Ivt 1: n ( x 10 


) J VT2iN (xl0- 2 ) 


1 


15.45 


21.56 


10.16 


9.30 


1.70 


1.56 


2 


21.56 


15.58 


9.30 


9.60 


1.56 


1.84 


3* 


15.58 


36.18 


9.60 


7.26 


1.84 


1.41 


4 


36.18 


15.92 


7.26 


11.26 


1.41 


1.73 


5* 


15.92 


25.92 


11.26 


8.17 


1.73 


1.56 


6 


25.92 


36.45 


8.17 


8.88 


1.56 


1.40 


7 


36.45 


34.24 


8.88 


9.40 


1.40 


1.38 


8 


34.24 


29.33 


9.40 


12.02 


1.38 


1.53 


9 


29.33 


34.47 


12.02 


11.65 


1.53 


1.38 


10 


34.47 


26.98 


11.65 


10.57 


1.38 


1.56 


11 


26.98 


23.39 


10.57 


10.16 


1.56 


1.64 


12* 


23.39 


29.46 


10.16 


8.36 


1.64 


1.50 


13 


29.46 


40.45 


8.36 


7.75 


1.50 


1.36 


14 


40.44 


25.84 


7.75 


8.11 


1.36 


1.50 


15 


25.84 


22.87 


8.11 


8.70 


1.50 


1.52 


16 


22.87 


39.42 


8.70 


7.42 


1.52 


1.37 



sure that the corresponding principal component drives all stocks approximately in the same way. Conversely, a highly 
localized eigenvector will cause the corresponding principal component to impact only very few variables (as a matter 
of fact eigenvectors related to sector eigenvalues are usually rather localized). In the light of these considerations, let 
us now see how the spectral analysis of the correlation matrices pj, and pj, can help to understand what causes the 
large spikes of the q ratio shown in Figure [3J 

In Tables IIIII and IIVI a few spectral quantities of interest related to the plots in Figure [3] are detailed. Namely, 
for each of the two sampling periods T% and T2 we report the values of the corresponding correlation matrix's largest 
eigenvalue (i.e. the market eigenvalues Ati,_/v and \t 2 ,n), the sum of the next three largest eigenvalues (i.e. the 

first sector eigenvalues), which we denote as A 7^ — 532= jv— 3 ^Ti,i and At 2 = Y2i=N-3 ^T 2 ,i> and the IPRs I VTi N and 
I Vt n of the eigenvectors corresponding to the largest eigenvalues. In both Tables, those samples where violations of 
the non-optimality bounds occur are highlighted with an asterisk *. A careful inspection of the spectral quantities 
reported in Tables [TTT1 and HVl allows to identify the co-occurrence of three specific phenomena in all such cases: 

• A very large relative increase of the market eigenvalue. 

• A decrease of the sum of the first few sector eigenvalues. 

• A decrease of the IPR of the eigenvector corresponding to the market eigenvalue (market eigenvector) . 

In Table [V] we report the relative variations of such quantities for the samples already highlighted in Tables IIIII and 
IIVI Also, for both the S&P and FTSE datasets, we report the largest positive variation of the market eigenvalue, 
and the largest negative variations of the sector eigenvalue contribution and of the market eigenvector's IPR for 
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TABLE V: Relative changes of the spectral quantities shown in Tables Hill and HVl Such values are reported for the highlighted 
samples of Tables [TTT1 and HVl corresponding to the ones violating the non-optimality bounds (see Figure Also, the largest 
positive variation in the market eigenvalue, and the largest negative variation in the sum of sector eigenvalues and IPR of the 
market eigenvalues are shown for the remaining (i.e. non-highlighted) samples. 



S&P Sample 


(At 2 ,jv 


— At 1 ,jv)/At 1 ,at 


(At 2 


-A Tl )/A n 




4* 




54.9% 




-9.2% 


-2.1% 


6* 




57.7% 




-27.0% 


-14.8% 


Others 




42.0% 




-26.8% 


-7.5% 


FTSE Sample 


(At 2 ,jv 


— A-TijivO/ATi.iV 


(At 2 


-a Ti )/a Ti 




3* 




132.2% 




-24.4% 


-23.4% 


5* 




62.8% 




-27.4% 


-9.8% 


12* 




26.0% 




-17.7% 


-8.5% 


Others 




72.4% 




-14.7% 


-10.4% 



the non-highlighted samples (labeled as "Others" in Table [Vj) . As can be seen, in all of the highlighted samples the 
aforementioned combination of relative variations is actually detected. Also, quite interestingly, the most pronounced 
variations, occurring for sample 3 of the FTSE dataset, appear to be responsible for the spectacular bound violation 
shown in the right plot of Figure [3J leading to a realized risk 40 times larger than the corresponding in-sample risk. 

Looking at the largest (positive or negative depending on the quantities being considered) relative variations oc- 
curring for samples other than the highlighted ones in Table [V] shows that samples leading to severe violations of the 
non-optimality boundaries are not necessarily characterized by the largest overall variations. As a matter of fact, it is 
of paramount importance to stress that only the co-occurrence of large relative variations for the three spectral quanti- 
ties mentioned earlier (the market eigenvalue, the sum of the first few sector eigenvalues and the market eigenvector's 
IPR) leads to the violation of the non-optimality bounds. From a financial viewpoint, a possible interpretation of 
such evidence is the following one. A sudden increase of the market eigenvalue going from the sampling period T\ to 
the sampling period T2, combined with the decrease of the first few sector eigenvalues, means that approximating the 
PC A equation p8[) to just one principal component over period T2, i.e. 

r T 2 ,t(t) ~ ^/Xt 2 ,n v^] n e T2 ,N{t), (19) 

becomes a much more reasonable assumption with respect to T\. Moreover, the increased derealization of the market 
eigenvector over Ti justifies the further approximation v^ N ~ 1/y/N, meaning, according to equation (1191) . that all 
stocks essentially follow the very same time evolution. This very rough, yet meaningful, approximation depicts all 
stocks as very strongly correlated, all equally driven by the market mode Ct 2 ,n- Correspondingly, all market sectors 
disappear. Clearly, such a picture would prevent from any chance of limiting risk, since a portfolio made by stocks 
evolving according to equation (|19[) is absolutely unprotected against collective downwards stock movements. So, all in 
all, the most relevant violations of the non-optimality bounds shown in Figure [3J happen whenever non-stationaritics 
in the stock dynamics cause the portfolio correlation matrix's largest eigenvalue to increase considerably and its 
eigenvector to delocalize almost completely, spreading evenly across all stocks. 

However, it is mandatory to notice that there does not seem to exist a cause-effect relationship between the changes 
in spectral quantities we mentioned and the violations of the non-optimality bounds for portfolio risk. As a matter of 
fact, just by carefully looking at Tables Hill and HVl one can identify a few cases (the most notable ones being sample 
9 of the S&P dataset and sample 16 of the FTSE dataset) where all of the previously discussed changes in spectral 
quantities actually occur, and the q ratio between realized and in-sample risk remains well within the expected region. 
So, loosely speaking, large relative changes in the market eigenvalue, in its eigenvector's localization properties, and 
in the first few sector eigenvalues appear to be necessary but not sufficient conditions for a large spike of the q ratio 
to take place. 

V. CONCLUSIONS 

The main goal of this paper was to study the effect of non-stationarities in financial time series on correlation 
coefficient measurements. The first part of this work was devoted to checking whether the common sense assumption 
that using longer time series provides more accurate Pearson correlation estimates actually matches financial empirical 
evidence. We verified that such common knowledge can be highly misleading from two different viewpoints. First, 
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we performed a global test by checking whether correlation estimates between the same pair of stocks over non- 
overlapping time windows are compatible with each other. Then, we performed a local test by checking whether the 
addition of new information, i.e. extending time series with the addition of new prices, actually improves the already 
available correlation coefficient estimates. In both tests we found clear evidence that non-negligible fractions of stock 
pairs and correlation estimates do not behave as intuition would predict. 

The second part of this work was then devoted to study the possible effects that relying on correlation estimates 
computed from non-stationary price processes might have on optimal portfolio selection. We relied on the very 
solid framework established in [13, [lij], where the effects of real-life portfolio selection were studied, highlighting the 
possible portfolio risk underestimations due to retaining optimal weights computed with past prices (i.e. with past 
correlation estimates) over the future. Such a framework allowed us to check for further risk underestimations due 
to biased correlation estimates affected by non-stationarities. Also in this case we detected (in real financial data) 
serious violations of the non-optimality bounds provided in (l3l . Il4j , and we provided a possible explanation for such 
evidence in terms of correlation matrix spectral properties. Namely, the largest violations of the non-optimality 
bounds were found in correspondence with large relative variations of the market eigenvalues, the most relevant sector 
eigenvalues and the market eigenvector's IPR. However, as pointed out at the end of Section Hvl the co-occurrence 
of such conditions does not always seem to be enough to cause violations of the non-optimality bounds of portfolio 
risk. Thus, a further investigation of other possible causes for portfolio risk underestimation, maybe not even related 
to non-stationarities, is absolutely in order in the next future. 

We are aware that this paper sounds a bit negative, being limited to show what might go wrong when empirically 
estimating correlation coefficients between stocks without providing any "healing recipe" to the problems we discuss. 
In this respect, it is worth remarking that in this paper we limited ourselves to measuring risk in terms of portfolio 
variance, in order to keep things as simple as possible. However, it is well-established that portfolio variance actually 
does not represent the most informative risk measure one can think of. It will then be a very interesting continuation 
of this work to repeat the portfolio analyses performed in Section IIVI on other, more refined, risk measures such as 
the Value at Risk 20] or the Expected Shortfall [2l| in order to test their robustness properties with respect to the 
changes in the portfolio correlation matrix spectral structure. Other future lines of research, which will be the topic 
of a forthcoming publication, will instead explore the possibility of understanding, and possibly predicting, financial 
crises in terms of correlation instabilities. 
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